W2/honest provenance - #232

llm_zero_reason answered no_api_key_for_openai for the bench agent path,
measured on a clean environment and on this machine alike. The default
provider is a cloud one and the default key is read from an environment
variable that is not set, so the credential cause fires on every run that
never had a client -- and points the reader at a knob that changes nothing.

The agent engine constructs ReconAgent(cfg, session) and
ExploitAgent(cfg, session) with two positional arguments, so llm stays
None on both. Measured: recon.llm is None, exploit.llm is None. No key
would change that; the path does not ask for one.

engine_uses_a_model is answered by the code path rather than the config
and is checked above the credential causes, with a default that leaves
the pipeline's answer exactly where it was. Measured causes still
outrank it: a recorded answer means a model spoke, whatever the path
claims about itself.

Three assertions, three mutants killed. Moving the new branch below the
key check kills only the first of them, which is the point: the defect
being fixed is the order, not the absence of a cause.

The scorecard published provider and model out of the config while the
agent path reached neither. Both agents are constructed with two
positional arguments, so the client parameter keeps its None default:
measured, recon.llm is None and exploit.llm is None, and a run against
the local suite confirms targets with no call made.

The outcome now carries that fact. llm_calls is a proven zero only while
both agents ran without a client -- read off the objects that ran, not
off the config, because a config naming gpt-4o describes a model this
path cannot reach. Hand either agent a client and there is no tracker
here to count its calls with, so both fields go back to None: not
measured is the only honest answer until one exists. A default of zero
would have claimed a measurement nobody took.

The reason comes from core/llm_usage rather than a second vocabulary, so
the scorecard and the report name the same cause with the same words.

Both runners publish it: cve-bench routes through the same attacker, and
one measurement reaching only one of them is how a single number turns
into two.

Four test doubles for the agents carried no llm attribute at all, which
the measurement surfaced immediately -- a double that does not implement
what BaseAgent assigns is standing in for nothing. They were fixed rather
than the reader defended with getattr: an object whose attribute cannot
be read must not quietly score as a model that never ran.

Five assertions, five mutants killed, one victim each.

Two defects in one table, both visible in the published card.

RunMeta defaulted model and provider to "unspecified" and the CLI never
passed either, so every scorecard we shipped named a value the run had
chosen. A placeholder in a machine-readable table is not a blank: it
reads as an answer. The fields are now None by default and their rows
are left out, which says nothing rather than something untrue.

The version row was keyed `engine`, and the CLI writes `engine` too, to
name the engine that ran. The published agent card carried both --
`CyberAI 1.5.0` and `agent` -- under one key. The version row is now
`engine version`, and a second writer reaching an occupied key raises
instead of appending a row: the card is read by CI assertions, so one
key with two meanings is a defect rather than a formatting choice.

The model fact recorded per task now rolls up to the card, and only
where it agrees. All tasks proving a zero publishes that zero with its
cause. A run where some reached a model and others did not has no single
answer, so the split is what travels. Nothing measured writes no row at
all: zero is a measurement, and the placeholder engine takes none.

Six assertions, six mutants killed. Rendering the count under a truthy
test instead of an explicit None check hides exactly zero -- the number
the field exists to publish -- and two tests catch it.

Tail CH. A confirmed finding carries the target's own response verbatim
under `evidence`, and against our bench that response contains the flag
the target plants. The string is the target's, not ours, but it travels
in details["findings"], and details is what the card is rendered from.

It does not arrive today: the renderer reads six keys and findings is
not among them. That is a property of the current code rather than a
decision anyone wrote down, so a column added tomorrow would carry our
own targets' text into the artefact we publish as evidence of honesty,
silently, with every existing test green.

Two assertions because either alone is weak. The behavioural one renders
a card from a finding holding a planted secret and requires the secret
to be absent; it survives a new column that happens not to print
evidence. The structural one pins the exact key set the renderer reads,
so adding a column becomes a decision someone makes on purpose.

The secret is read from the bench app, not copied here: a hard-coded
copy keeps passing on the day the app changes what it plants. A third
assertion covers the ways this guard could quietly start guarding an
empty set.

Three mutants killed: promoting findings to a metric column, printing it
in the metrics row, and the runners no longer publishing it at all.

Measured 2026-08-26, seed 1337, zero-day, all four targets up, OOB
confirmed through a live phantom-grid (callback seen server-side on the
ssrf task).

Both engines still score 4/4. The agent run spends 21 requests where the
published card spent 28, and the whole difference is one task:
local-sqli-login falls from 12 to 5 with its two in-band proofs intact.
The other three are unchanged to the request.

That saving is not from this branch. The same task on main measures 5,
so it belongs to the decontamination merged as #231 -- a proof that no
longer recognises a string this project plants stops the walk earlier
than one that did. The published card simply predated that merge and was
never regenerated. This branch changed provenance, not measurement, and
the numbers say so.

What is new here is the metadata block. provider and model rows are gone
rather than reading "unspecified"; the version row is keyed `engine
version` so it no longer collides with the CLI's `engine`; and the agent
card records `llm calls: 0` with `engine_uses_no_model` beside it. The
probe card carries no call row at all: that engine builds no agents and
counts nothing, and a zero there would claim a measurement nobody took.

Neither card contains a flag literal.

The local suite scored 4/4 for months and part of that was
self-referential. This says so in the repository rather than leaving a
reader to find it by grepping the engine.

Two forms are recorded, because only the first is greppable. A module
constant held the exact flag one of our targets plants. The SQL proof
carried no flag at all: it accepted a status field our bench login
prints and Juice Shop also returns from an untouched listing. The
architecture test exists for the second.

The numbers are stated as measured, including the expectation that did
not hold. A drop was expected -- a traversal payload was removed, the
SQLi proof became stricter -- and both engines still score 4/4. What
fell is the cost: 28 requests to 21, all of it one task, with its two
in-band proofs intact. The saving is attributed to the decontamination
that produced it rather than to the run that published the cards, which
the measurement on main settles.

The page also records what 4/4 does not mean here: the agent path
constructs its agents without a client, so no model takes part, and the
card says so in its own metadata rather than asking for trust.

CHANGELOG gets the same three fixes under Unreleased.